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(54) Method for the acquisition of images by confocal microscopy 



(57) A method for the acquisition of images by 
means of confocal microscopy In which a new Image Is 
calculated which is constituted by the "maxima" among 
the corresponding elements of each captured image. 
The new Image contains mainly the signal coming from 
the most luminous and in focus areas and even the sig- 
nal coming from the less luminous areas, as being out 
of focus or laterally displaced with respect to the grid 
positions. A further image is then calculated which Is 
constituted by the "minima" among the corresponding 
elements of each captured image and the requested 
confocal Image is then obtained by calculating the dif- 
ference between the "maxima" Image and the "minima" 
Image. 
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Description 

The present invention relates to a method for the 
acquisition of images by means of confocal microscopy. 

More particularly, the present invention relates to a s 
method for the confocal images acquisition in optical 
systems for the analytical microscopy and in other opti- 
cal systems with a relatively high numerical aperture in- 
volving the electronic images acquisition. The objects to 
be analysed can be absorbing (transmission), reflecting 10 
or fluorescent, and the optical arrangements can there- 
fore vary among those for the analysis under transmitted 
or reflected light or under fluorescence. 

The confocal microscopy or anyway the analysis 
and the processing of images of objects by means of '5 
confocal systems with optical and electronical means, 
is a technique which grew considerably in recent years 
only. On this subject reference can be made to the article 
"Confocal Microscopy" by Jeff W. Lichtman, Scientific 
American, August 1 994, as well as patent applications 20 
DE-A-4023650 published on January 30, 1992, EP-B- 
0466979 published on October 5. 1994 and other doc- 
uments. 

Confocal techniques are very effective when the rel- 
atively thick structure(s) under obscn/ation, of which a 25 
tridimensional structure must be obtained, are im- 
mersed in a relatively transparent and thin material. 

When the structures to be analysed by a confocal 
method are in a relatively thick material and with light 
scattering properties, an increasing loss of contrast with 30 
the increase of the scattering occurs that may result in 
a poor observability or, in an extreme case, the inob- 
servability of the interested microscopical structures. 

According to the known art, the object of interest is 
illuminated by means of a group of light beams concen- 3S 
trated on positions belonging to a focus plane and ar- 
ranged on said plane according to a ordered bidimen- 
sional grid. The complete lighting of the field under ex- 
amination is obtained by the systematical displacement 
according two coordinates of the grid so as to cover all 40 
spaces among the focused light points originated by the 
grid. 

According to the present invention, in order to carry 
out the acquisition of a confocal image, the acquisition 
of the set of partial images, obtained in correspondence 
to each position of the illumination grid in the focus plane 
is first carried out by means of an electronic image sens- 
ing device. In this way, a values distribution depending 
on the grid position is obtained for each image element. 

According to the present invention confocal images ^o 
are obtained by analysing the above mentioned values 
distribution by using statistical-mathematical proce- 
dures with an increasing degree of complexity and ac- 
curacy, of the type described below 

In particular, referring to a monodimensional exam- 55 
pie for sake of simplicity, according to the present inven- 
tion a new image constituted by the maxima of luminos- 
ity among the corresponding elements of each captured 
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image is calculated. The image calculated in this way, 
Max(x), has confocal characteristics as mainly contains 
the signal coming from the most luminous and in focus 
areas, even if a part of the signal coming from the less 
luminous areas is contained as they are relatively out 
of focus or sideways displaced with respect to the grid 
positions. 

Still according to the invention a second image, Min 
(x), constituted by the minima of luminosity among the 
corresponding elements of each captured image is then 
calculated. The second image mainly contains the sig- 
nal coming from the less luminous areas in so far as 
they are less in focus or displaced sideways, while it 
tends to exclude the signal coming from the most lumi- 
nous areas. 

Consequently, the image of the luminosity maxima 
contains both the lighting due to the part in focus of the 
object under examination, and that of the less luminous 
part, not in focus, while the minima image tends to be 
affected only by the less luminous part and out of focus 
of them. As a consequence, according to the invention 
an image with enhanced confocal characteristics t.e a 
more accurate optical section, is obtained by calculating 
the difference between the maxima image and the mini- 
ma image. As a matter of fact, this allow to increase the 
contrast and to reduce the scattering effect of the ma- 
terial in which the object under examination is im- 
mersed. 

In case of noise the above operation tends to shift 
the zero level. A further improvement of the invention, 
fit for avoiding the above shift of the zero level and even 
to increase the spatial resolution, consist in calculating 
a new image, Med(x), formed by the median values or 
by the average values among the corresponding ele- 
ments of each of the captured images, and in carrying 
out a mathematical processing of the signal that can be 
represented by the formula. 

K [Max(x) -1- Min(x) - 2 Med(x)] 

where K is a gain factor correlated to the shape of the 
lighting areas. 

The characteristics and the advantages of the 
method according to the present invention will be appar- 
ent from the following description of an actually pre- 
ferred embodiment made as a not limiting example with 
reference to the attached drawings, in which: 

Figure 1 is a general exemplifying diagram of a con- 
focal microscopy system; 

Figure 2 is a wave form diagram of the signals of a 
system of light reflection microscopy of the image 
on a generic line of a matrix sensor as a function of 
the displacement of the illumination grid along an 
axis; 

Figure 3 is a diagram similar to that of figure 2 in 
which the result of the "maximum minus minimum" 



2 



BNSDOCID: <EP 0e331B1A1_l > 



EP0 833 181 A1 



3 

operation is shown 
Figure 4 is a diagram similar to those of figures 2 
and 3. in which the operation K (Max + Mtn - 2Med) 
is executed: 

Figure 5 is a possible data processing block flow 
diagram for the system of the invention: 
Figure 6, particulars a), b). c), c). e). f). shows var- 
ious images from the initial condition to the final one 
in an image processing method as shown in figure 
4. 

With reference to figure 1 the conventional general 
arrangement of a confocal microscope is shown herein. 
A light source 10 cooperates with a collimating lens 11 , 
the output light beam of which passes through a hole 
matrix diaphragm 12 or equivalent device which is dis- 
placed according two coordinate axes x, y by means of 
motors, not shown, preferably by step motors. The 
structure formed by said hole matrix diaphragm can be 
replaced by a xy matrix scanning device formed by a 
liquid crystal light valve optoelectronic device and with- 
out moving mechanical parts. The light passing through 
hole matrix diaphragm 12 meets a beam divider 15. A 
part of the light crossing beam divider 15 is focused on 
a specimen 16 by means of a lens 17 The part of the 
light reflected at left of beam divider 1 5 is captured by a 
light trap 19 as well-known in these optical arrange- 
ments to eliminate back scattered disturbing light. 

The light returning from specimen 16 is directed to 
an image photoelectric sensor 18 such as a bidimen- 
sional CCD sensor. The operation of such an optical ar- 
rangement is well-known to a person skilled in the art 
and a detailed description is omitted. 

As stated above, the optical system shown in figure 
1 is relevant to the analysis by reflection of a specimen 
and an electromechanical system for the displacement 
of the hole matrix diaphragm 12 is provided. Different 
optical systems relevant to the specimen analysis by 
transmission and/or by fluorescence are included in the 
scope of the invention. In particular it is included in the 
scope of the invention the use of optoelectronic means 
to carry out the scanning of the hole matrix diaphragm 
12 or to perform in a different way the function of the 
illumination grid. This can be done, tor example, by 
means of spatial light modulators without moving parts 
of the type disclosed by Fairfield and Vaytek in VI RTU AL 
MICROSCOPE, European Microscopy, May 1996. The 
same operation can also be carried out by using DMD 
(Digital Micromirror Device) devices as described by J. 
M. Younse in "MIRRORS ON A CHIPV IEEE Spectrum, 
November 1993. 

The operation of the system according to the inven- 
tion will now bo described. With reference to figures 2, 
3 and 4 relate to a single scanning line through a generic 
specimen and are equal to those which could corre- 
spond to linear image sensor or to a generic scanning 
line passing through the specimen of a bidimensional 
image sensor. It has to be pointed out that the use of an 



image sensor of a linear or bidimensional type is given 
only by way of example of a commonly used optoelec- 
tronic sensor, and it is understood that the use of sen- 
sors of different type is included in the scope of the in- 
s vention. 

With reference in particular to figure 2, the horizon- 
tal axis represents the space that the specimen crosses, 
while the vertical axis represents relative intensity (or 
density) of the signal on an arbitrary scale in the various 
TO figures. 

Line 100 indicates the confocal illumination process 
of the specimen carried out by the displacement, for ex- 
ample, along the x-coordinate of the hole matrix dia- 
phragm or spatial modulator 12, as shown in figure 1 . In 
line 100 peaks 101 are the illumination peaks of the 
specimen in a generic position m of diaphragm 12. 
Peaks 102, 103 are the displacement of peaks 101 in 
the positions m+1 , m-f2 respectively controlled by the 
motors that perform the displacement of diaphragm 12. 

20 In the representation of figure 2 (and also of figures 
3 and 4) the "density" of the peaks is simplified and re- 
duced with respect to the real condition. 

Line 105 shows the wave form diagram of the opti- 
cal density (opacity) of a generic specimen, the path 

2S comprising portions 1 06, 1 07 having null optical density 
(perfect transparency) of the specimen and portions 
108, 109, 110, 111, 112 having an intermediate optical 
density between the null density (1 06, 1 07) and the max- 
imum one 113. 

30 Line 14 shows the output of a detector such as a 
CCD photodetector (indicated at 1 8 in figure 1 ), said out- 
put being formed by an envelope of peaks correspond- 
ing to the peaks of line 100 spatially modulated by the 
optical density of the specimen shown in line 105. 
35 tf the light peaks of line 114 in correspondence to 
the scanning steps (x-axis) are extracted with a conven- 
tional data processing method, envelope 1 1 5 of line 1 1 6 
is obtained representing a true reproduction of the spec- 
imen image details exemplified at line 105. 
^0 In figure 3 there is shown the signal processing ac- 
cording to the invention in which an unwanted back- 
ground signal, of a systematic type and produced by the 
presence of something causing scattering in the speci- 
men not being part of the interested image of the same, 
^5 is removed. 

Lines 1 00 and 1 05 of figure 3 have the same mean- 
ing as the corresponding one of figure 2. 

In the case of figure 3, an unwanted background 
signal or systematic noise indicated at line 120 is asso- 
so ciated to curve 105 representing the specimen. 

As a consequence of the photosensor output signal 
indicated at line 114', the 'minima" of the specimen sig- 
nal will be raised with respect to the base line of an 
amount corresponding to the background signal, which 
55 is shown alone by line 121 formed by drawing out the 
"minima" of image signal 114'. 

Unless the operation of the invention would be ear- 
ned out, the output signal would be that indicated at line 
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116' constituted by an envelope practically formed by 
adding the specinnen signal (line 105) to the unwanted 
signal (line 120). 

By subtrBcting signal 121 from signal 116' fine*! -*^iG' 
nal 1 22 is obtained. Final signal 1 22 can be represented 
as A(Max - Min) where A is a proportionality constant 
that can be taken equal to 1 . 

It will be appreciated that the signal of line 122 is 
"cleared" of systematic noise 120, thus removing possi- 
ble artifacts in the final image and improving the image 
contrast. 

A variation of the processing method according to 
the invention, directed to remove (or. at least, signifi- 
cantly reduce) systematic and/or casual noises and the 
like is shown in figure 4. 

Systematic noise have been previously defined. 
Casual noises can have vanous origin, for exannple a 
noise in the photoelectric sensor of the CCD detector, 
casual fluctuations of the tight source (for example, cas- 
ual instability of the arc light source, etc.). 

Line 1 30 of figure 4 shows the illumination (likewise 
line 100 of figure 2) . Line 131 shows the specimen den- 
sity (likewise line 1 05 of figures 2 and 3). Line 1 32 is the 
theoretical image of the specimen taken alone. 

Line 1 33 shows the path of the possible systematic 
+ casual noise. Line 1 34 shows the photoelectric sensor 
output signal, curve 135 the envelope of the "maxima" 
and line 1 36 the envelope of the "minima". Line 1 37 rep- 
resents an average of the "maxima" and "minima" and 
line 138 represents the function K (Max + Min - 2 fV/Ied) 
representing the final component coming from the sys- 
tem and constituting the final image 

In other words, during the scanning operation, ac- 
cording to I. the position of the illumination peaks chang- 
es. A set of vectors lj(x), numerically reproducing the 
intensity profile detected from the specimen, according 
to X, is thus obtained. 

From li(x) set the following vectors are obtained: 

=> Max(x): formed by the "maxima" of the corre- 
sponding elements being part of vectors li(x) when 
i varies (figures 4, line 135); 

=> Min(x), formed by the "minima" of the corre- 
sponding elements being part of vectors lj(x) when 
i varies (figure 4, line 136); 

=> Med(x). formed by the "averages" of the corre- 
sponding elements being part of vectors lj(x) when 
i varies (figure 4, line 137). 

From the above vectors the following results can be 
obtained: 

A) Max(x) 

It detects the amplitude of the confocal signal 
in the presence of thin objects and in the presence 
of noises. The presence of systematic noise (back- 
ground, thick object, etc.) and the amplitude of the 
casual noise sum up to the signal. 
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B) Max(x) - Min(x) 

It delects the amplitude of the confocal signal 
even in the presence of systematic noise (back- 
ground, thick object, etc.). The amplitude of the cas- 
5 ual noise is still summed up. 

C) K [Max(x) + Min(x) - 2Med(x)l 

It detects the amplitude of the confocal signal 
correctly, even in the presence of systematic or cas- 
ual noise. It requests a gain factor K, which is a f unc- 
10 tion of the shape of the illumination areas. If R is the 
modulation rate dark/light, then K=(R+1 )/(R-1 ). 

A block flow diagram of the algorithm, which, as an 
example, implements the operation mentioned above 

15 and shown in figure 3, is shown in figure 5. With refer- 
ence to figure 5, being X the number of elements of a 
monodimensional image detector and N the number of 
displacements needed to cover all the spaces between 
the lighting points originated by the illuminating dia- 

20 phragm, the flow chart shows the complete procedure 
to obtain a monodimensional confocal image. 

Setting the indexes n and x to zero the first image 
Iq(x) is captured and the vectors Max(x), Min(x) and Med 
(x) are initialized directly copying the vector l^j(x) into 

25 them, this operation being made for all x until the con- 
dition x<X is verified. 

The index n is incremented, this corresponding to a 
further position of the diaphragm. The index x is set to 
zero again and a new image is acquired. The elements 

30 of the vectors Max(x) and Min(x) are compared, with re- 
spective elements of lr,(x), and conveniently updated. 
The calculation of the vector f^ed(x) follows. The loop 
continues until both the conditions x<Xand n<N are ver- 
ified. 

35 Finally, confocal images can be obtained by the fol- 
lowing expressions: 

a) l=Max(x) b) l=:Max(x) - Min(x) 
c) l=K(Max(x) + Min(x) - 2Med(x)) 

40 

In the particulars a), b), c), d), e) and f) of figure 6 
photographs of images corresponding to the above 
mentioned operations, identified by a relevant legend, 
earned out in the presence of a real microscopic object 

45 are shown. Figure 6 is not described in further detail as 
deemed self-explanatory for a person skilled in the art. 

The present invention has been described with ref- 
erence to an actually preferred embodiment and exem- 
plified with reference to reflexion microscopy and a spa- 

50 tial modulator of mechanical type, but a person skilled 
in the art will understand that the object of the invention 
can be also applied to transmission systems or fluores- 
cence systems and even to systems using spatial mod- 
ulators of the optoelectronic type without moving me- 

55 chanical parts. All these variations are deemed to be en- 
compassed in the scope of the invention as set forth in 
the appended claims. 
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Claims 



A method for the acquisition of images in the field 
of confocal microscopy, in which the object to be 
analysed is illuminated by means of a group of light 
beams concentrated on positions being part of a fo- 
cus plane and arranged on saic plane according to 
an ordered grid, the complete illumination of the ar- 
ea under examination being carried out by means 
of the systematic displacement of said grid, so as 
to cover all the interposed spaces, and in which the 
acquisition of a confocal optical section is carried 
out first of all by the acquisition, through an image 
electronic sensor, of the set of the partial images 
obtained in correspondence of each position of said 
illumination grid on the focus plane, said method be- 
ing characterized in that a new image is calculated 
constituted by the maxima among the correspond- 
ing elements of each captured image: 



[Al 



confocal image = r^ax(x). 
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The method according to claim 1 , in which, in addi- 
tion to said imago containing mainly the signal com- 
ing from the most luminous and in focus areas and 
even the signal coming from the less luminous ar- 
eas, as being out of focus or sideways arranged 
with respect to the grid positions, a further image is 
calculated constituted by the "minima" among the 
corresponding elements of each captured image, 
said further image containing mainly the signal 
coming from the less luminous areas, as being out 
of focus or sideways arranged, while tending to ex; 
elude the signal coming from the most luminous ar- 
eas, so that the "maxima" image contains both the 
information due to the in focus part of the object and 
that duetto the less luminpus part, as being out of 
focus, while the "minima** image tends to be affected 
only by the less luminous and out of focus thereof, 
said method being characterized in that the wanted 
confocal image, i,e. an optical section, is obtained 
by calculating the difference between the "maxima" 
image and the "minima" image according to the for- 
mula: 



25 



30 



35 



formed by the average values or median values of 
the light distribution relevant to each element of the 
captured images when varying the position of the 
illumination grid. 

4. The method according to the previous claims, char- 
acterized in that a digital processing of the signal is 
carried out as set forth in the block flow diagram 
shown in figure 5. 

5. The method according to the previous claims, cha- 
racaterized in that it is applied to confocal systems 
of the reflexion or transmission or fluorescence 
type. 

6. The method according to claim 5. characterized in 
that a ordered grid light source or spatial modulator 
comprises a scanning optoelectronic system with- 
out moving mechanical parts. 

7. The method according to claim 6, characterized in 
that said illumination scanning system comprises a 
liquid crystal matrix light valve or other optoelectric 
device. ^• 

8. The method according to the previous claims, char- 
acterized in that the tridimensional analysis of a 
specimen is carried out by varying the focusing of 
the system or displacing the specimen along the op- 
tical axis so as to obtain a prefixed number of optical 
sections through all the thickness thereof. 

Apparatus for confocal microscopy operating under 
reflexion or transmission or fluorescence using the 
method as claimed in any of the previous claims. 
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[B] 



confocal image = Max(x) - Min(x). 



3. The method according to the claims 1 or 2, charac- so 
terized in that said wanted confocal image can be 
expressed by the formula: 

[C] confocal image = K[Max(x) - Min(x)-2fy/ed(x)], 

K being a gain factor which is a function of the shape 
of illumination areas and Med(x) being a vector 
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